-
Notifications
You must be signed in to change notification settings - Fork 12.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove Nonterminal
and TokenKind::Interpolated
#124141
base: master
Are you sure you want to change the base?
Remove Nonterminal
and TokenKind::Interpolated
#124141
Conversation
Some changes occurred in src/tools/rustfmt cc @rust-lang/rustfmt |
42a623a
to
c133e16
Compare
❤️ @nnethercote for working on this. Thank you! I'm not sure if there's a way for me to help, as someone who doesn't really know much about the compiler innards, but please LMK if you think of something. |
@ijackson: thanks! I'm curious why you are interested in this change, given that it's a compiler internals rearrangement? |
After this is done |
Yes. |
Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.
c133e16
to
7aef5db
Compare
This comment has been minimized.
This comment has been minimized.
Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.
…, r=<try> Print `token::Interpolated` with token stream pretty printing. This is a step towards removing `token::Interpolated` (rust-lang#124141). It unavoidably changes the output of the `stringify!` macro, generally for the better. r? `@petrochenkov`
#125174 carves off a piece of this PR so it can be merged separately. |
☔ The latest upstream changes (presumably #123865) made this pull request unmergeable. Please resolve the merge conflicts. |
Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`
Rollup merge of rust-lang#125236 - dtolnay:expandtest, r=nnethercote Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`
Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for all the `c2*` macros. The AST pretty printer now has more thorough testing thanks to rust-lang#125236. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.
It's great to see that Of course it prevents a lot of interesting stuff like reparsing |
How hard would it be to get this to a perf run? |
…<try> rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal`. r? `@petrochenkov`
☔ The latest upstream changes (presumably #132831) made this pull request unmergeable. Please resolve the merge conflicts. |
It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.
9dfb022
to
313dbcd
Compare
It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.
It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.
…petrochenkov rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal` (rust-lang#124141). r? `@petrochenkov`
…petrochenkov rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal` (rust-lang#124141). r? `@petrochenkov`
…petrochenkov rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal` (rust-lang#124141). r? `@petrochenkov`
It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.
We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`.
Notes about tests: - tests/ui/parser/macro/trait-object-macro-matcher.rs: the syntax error is duplicated, because it occurs now when parsing the decl macro input, and also when parsing the expanded decl macro. But this won't show up for normal users due to error de-duplication. - tests/ui/associated-consts/issue-93835.rs: ditto. - The changes to metavariable descriptions in this PR's earlier commits are now visible in error message for several tests.
The one notable test change is `tests/ui/macros/trace_faulty_macros.rs`. This commit removes the complicated `Interpolated` handling in `expected_expression_found` that results in a longer error message. But I think the new, shorter message is actually an improvement. The original complaint was in rust-lang#71039, when the error message started with "error: expected expression, found `1 + 1`". That was confusing because `1 + 1` is an expression. Other than that, the reporter said "the whole error message is not too bad if you ignore the first line". Subsequently, extra complexity and wording was added to the error message. But I don't think the extra wording actually helps all that much. In particular, it still says of the `1+1` that "this is expected to be expression". This repeats the problem from the original complaint! This commit removes the extra complexity, reverting to a simpler error message. This is primarily because the traversal is a pain without `Interpolated` tokens. Nonetheless, I think the error message is *improved*. It now starts with "expected expression, found `pat` metavariable", which is much clearer and the real problem. It also doesn't say anything specific about `1+1`, which is good, because the `1+1` isn't really relevant to the error -- it's the `$e:pat` that's important.
This involves replacing `nt_pretty_printing_compatibility_hack` with `stream_pretty_printing_compatibility_hack`. The handling of statements in `transcribe` is slightly different to other nonterminal kinds, due to the lack of `from_ast` implementation for empty statements. Notable test changes: - `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group around a metavar.
This time when converting them to proc-macro `Group` form.
Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.
Notes about tests: - tests/ui/rfcs/rfc-2294-if-let-guard/feature-gate.rs: some messages are now duplicated due to repeated parsing. - tests/ui/rfcs/rfc-2497-if-let-chains/disallowed-positions.rs: ditto. - `tests/ui/proc-macro/macro-rules-derive-cfg.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group around a metavar.
`NtBlock` is the last remaining variant of `Nonterminal`, so once it is gone then `Nonterminal` can be removed as well.
They are no longer needed. This does slightly worsen the error message for a single test, but that test contains code that is so badly broken that I'm not worried about it.
313dbcd
to
2479f6c
Compare
The job Click to see the possible cause of the failure (guessed by this bot)
|
The first rule of the `features` macro looks like this: ``` macro_rules! features { ( @target: $target:ident; @cfg: $cfg:meta; @MACRO_NAME: $macro_name:ident; @MACRO_ATTRS: $(#[$macro_attrs:meta])* $(@BIND_FEATURE_NAME: $bind_feature:tt; $feature_impl:tt; $(#[$deprecate_attr:meta];)?)* $(@NO_RUNTIME_DETECTION: $nort_feature:tt; )* $(@feature: #[$stability_attr:meta] $feature:ident: $feature_lit:tt; $(without cfg check: $feature_cfg_check:literal;)? $(implied by target_features: [$($target_feature_lit:tt),*];)? $(#[$feature_comment:meta])*)* ) => { ``` Notice all the `tt` specifiers. They are used because they are forwarded to another macro. Only `ident`, `lifetime`, and `tt` specifiers can be forwarded this way. But there is an exception: `$feature_lit:tt`, which was added recently. In theory it should cause an error like this: ``` error: no rules expected `literal` metavariable --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:54:91 | 51 | / macro_rules! $macro_name { 52 | | $( 53 | | ($feature_lit) => { 54 | | $crate::detect_feature!($feature, $feature_lit $(, without cfg check: $feature_cfg_check)? ... | | ^^^^^^^^^^^^^^^^^^ no rules expected this token in macro call ... | 88 | | }; 89 | | } | |_________- in this expansion of `is_x86_feature_detected!` | ::: std/tests/run-time-detect.rs:145:27 | 145 | println!("tsc: {:?}", is_x86_feature_detected!("tsc")); | ------------------------------- in this macro invocation | note: while trying to match keyword `true` --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:12:55 | 12 | ($feature:tt, $feature_lit:tt, without cfg check: true) => { | ^^^^ = note: captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens = note: see <https://doc.rust-lang.org/nightly/reference/macros-by-example.html#forwarding-a-matched-fragment> for more information ``` (The URL at the end of the error has more details about this forwarding limitation.) In practice it doesn't cause this error. I'm not sure why, but the existing macro implementation in rustc is far from perfect, so it's believable that it does the wrong thing here. Why does this matter? Because rust-lang/rust#124141 is modifying the macro implementation, and when that PR is applied the error *does* occur. (It's one of several cases I have found where the existing compiler accepts code it shouldn't, but #124141 causes that code to be rejected.) Fortunately the fix is simple: replace the `literal` specifier with `tt`.
This CI failure:
is due to a new issue in stdarch. There is some code in stdarch that the current compiler accepts but shouldn't, and this PR causes it to fail to compile. Details in rust-lang/stdarch#1680. |
Remove `NtVis` and `NtTy` The next part of rust-lang#124141. The first actual remove of `Nonterminal` variants. `NtVis` is a simple case that doesn't get much use, but `NtTy` is more complex. r? `@petrochenkov`
A third attempt at this; the first attempt was #96724 and the second was #114647.
r? @ghost